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Executive Summary 


The project 


Catch Up® ' Numeracy is a one to one intervention for learners who are struggling with numeracy. It 
consists of two 15-minute sessions per week, delivered by teaching assistants (TAs). The approach is 
based on research indicating that numeracy is not a single skill, but a composite of several component 
skills that are relatively discrete. The intervention breaks numeracy down into ten elements, including 
counting verbally, counting objects, word problems and estimation. Pupils are assessed on each 
component and instruction is targeted on those areas requiring development. 


In this evaluation, the intervention was run for 30 weeks and delivered to Year 2-6 pupils who were 
struggling with numeracy, as identified by TAs. The Catch Up Numeracy intervention was compared to 
a ‘business as usual’ control group and a ‘time equivalent’ intervention group, who received the same 
amount of one to one teaching by TAs, but did not use Catch Up Numeracy. Those TAs delivering 
Catch Up Numeracy were supplied with detailed session plans and received three half-day training 
sessions, led by Catch Up and Dr Ann Dowker of the University of Oxford. The project ran from 
September 2012 to July 2013. 


What impact did it have? 


The overall effect size of Catch Up Numeracy in comparison to the ‘business as usual’ control group 
was +0.21, meaning the programme led to a noticeable improvement in numeracy outcomes. This 
effect size suggests that, on average, pupils receiving the interventions would make approximately 
three additional months of progress over the course of the year compared to pupils that did not. 


However, the ‘time equivalent’ group also showed similar significant gains (+0.27), suggesting the 
effect is likely to be a result of regular and sustained one to one teaching, rather than an intrinsic 
benefit of Catch Up Numeracy. 


The study demonstrates that one to one teaching with TAs is an effective strategy to increase 
numeracy skills in Year 2-6 pupils. Sub-group analysis did not identify any differential effects for pupil 
gender or eligibility for free school meals. 


The process evaluation indicated that most TAs valued Catch Up Numeracy and believed it had a 
positive impact on pupils’ confidence, engagement with learning and willingness to attempt numeracy 
problems. 


Catch Up Numeracy 
vs. control 


i sical 


102 


Equivalent time one 
to one support vs. 
control 


+0.27 +4 0.06-0.49 


KKK 


*Evidence ratings are a new measure under development based on a number of factors including study type, size and drop-out. 
Ratings are provisional and are not given for sub-group analyses, which will always be less secure than overall findings. 
For more information about ratings visit: www.educationendowmentfoundation.org.uk/evaluation. 


' Catch Up@ is a not-for-profit UK registered charity (1072425). Catch Up® is a registered trademark 
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How secure is this finding? 


The evaluation was set up as an effectiveness trial to test the impact of Catch Up Numeracy in 
comparison to a ‘business as usual’ control group and an ‘equivalent time’ intervention group, with the 
developer leading the training and overseeing the provision of the intervention. Effectiveness trials aim 
to test whether an intervention can work at scale, in a large number of schools. 


The findings are based on a three-arm randomised controlled trial using an intent to treat analysis (i.e. 
pupils were compared in the groups to which they were originally randomly assigned). Six pupils from 
each of 54 primary schools (two with two sites within the same school) were randomly assigned to one 
of three groups: a control group that received normal teaching, a Catch Up Numeracy intervention 
group that received the intervention as described above, and an ‘equivalent time’ group that received 
two 15 minute sessions a week without Catch Up Numeracy, to replicate the one to one nature of the 
intervention. The primary outcome measure was numeracy ability, as measured by the Basic Number 
Screening test. Blind marking of test papers was undertaken. There was relatively low drop out, 
relatively evenly spread across the control and intervention groups. 


The main threat to the internal validity of this trial is the possibility that TAs delivering Catch Up 
Numeracy passed on knowledge of the intervention to those TAs in the ‘time equivalent’ group within 
the same school — called cross-contamination. There is some evidence that the time equivalent group 
of TAs had some knowledge of Catch Up Numeracy and amended their approach in light of this 
knowledge, although it is unclear as to whether this had an impact on the results of the trial. Overall, 
the evaluators consider that the effect of one-to-one teaching is robust. However, the differences 
between the time equivalent group and the Catch Up group are harder to identify. The study findings 
are consistent with the wider evidence base on one to one tuition, and a smaller number of studies 
evaluating the use of TAs for one to one support. 


The process evaluation revealed that there was some variance in the way in which the intervention 
was delivered, including a failure to deliver two 15-minute sessions each week for the full 30 weeks, 
as the trial intended. It is suggested successful implementation would benefit from TAs having 
sufficient time to plan and prepare for the sessions, with time scheduled specifically within the existing 
timetable. 


How much does it cost? 


The cost of the approach is estimated at £130 per pupil. This estimate includes resources (estimated 
at £2.00 per pupil), direct salary costs of TA (£95 per pupil), initial training (£17.50 per pupil) and on- 
going monitoring and support (£8.75 per pupil). Estimates are based on a school delivering the 
intervention to 40 pupils and training two TAs and one teacher as the Catch Up Coordinator who 
supports the TAs but does not work directly with pupils. 


1. Within this trial, one-to-one support by TAs led to a significant gain in numeracy skills. 


2. Catch Up makes similar significant gains, but there is little evidence that Catch Up Numeracy 
provided any additional gains in numeracy outcomes over and above those from one to one 
teaching itself. 


3. Schools can find it challenging to run two 15 minutes sessions per week, due to timetabling and 
other issues. 


4. Structured interventions, such as Catch Up Numeracy, should be planned into the timetable from 
the start of the new school year to ensure they are given priority and status. 
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Introduction 


Intervention 


Catch Up Numeracy is a one-to-one intervention for primary school children who are struggling with 
numeracy. It consists of two 15-minute sessions per week, usually delivered by TAs. To prepare them 
for the intervention, classroom assistants are supplied with detailed lesson plans and receive three 
half-day training sessions. The intervention breaks numeracy down into ten components, tests 
children’s ability on each component and targets subsequent instruction so that the tutor always 
addresses the exact area requiring development. Components include counting verbally, counting 
objects, reading and writing, hundreds, tens and units, ordinal numbers, word problems, estimation, 
remembered facts and derived facts and translation. 


The approach is based on research indicating that numeracy is not a single ‘big’ skill, but a compound 
of several ‘little’ skills that seem to be quite discrete. Children (and adults) may be very strong in some 
skills but very weak in others. Further, brain-imaging studies suggest that the different skills are 
handled by different parts of the brain. By recognising and building on this finding, the Catch Up 
Numeracy intervention enables tutors to diagnose and treat problems precisely and effectively. 


Background evidence 


The Catch Up Numeracy approach is informed by a number of key research findings on how children 
learn mathematics. The first is that arithmetical cognition is made up of multiple components, and that 
there can be large discrepancies in ability on each (Dowker, 1998; 2005). The second is that different 
arithmetical procedures are carried out in different parts of the brain (e.g. Castelli et a/., 2006). The 
third is that the different components of arithmetical cognition cannot be arranged into a strict 
hierarchy: a child may be strong on a component that most struggle with, and weak on a component 
that most find easy (Denvir and Brown, 1986). 


The Catch Up Numeracy intervention was designed with these findings in mind. The initial diagnostic 
tests identify exactly which components or processes a child struggles with; tutors then direct their 
attention to these areas, rather than always starting with what seems the most basic process in the 
sequence. 


A trial of Catch Up Numeracy was carried out in 2010 with the participation of 246 pupils across 
eleven local authorities (Dowker and Sigley, 2010). Schools were asked to nominate four to six 
children for the trial, assigning each pupil to either a treatment group (who received Catch Up 
Numeracy), a matched-time control group (who received the same amount of one-to-one maths 
instruction, but not using the Catch Up methods), or a pure control group (who received classroom 
teaching as normal). Assignment to experimental group by the school rather than through 
randomisation may have introduced some bias into the trial, but it is impossible to say what direction 
this might be in. 


Children in the trial were tested before and after using the Basic Number Screening Test. There were 
small differences between groups on pre-test scores, but these were not statistically significant. Post- 
test results suggest the intervention had medium to large effects: 0.3 standard deviations versus 
matched-time controls, and 0.4 standard deviations versus pure controls. These results were found to 
be highly statistically significant. 
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Overall, the research behind Catch Up Numeracy is quite strong. The intervention is rooted in an 
evidence-based understanding of how children learn arithmetic, and the only trial so far has shown 
promising results. The logical next step is a full randomised trial that includes enough pupils and TAs 
to detect educationally relevant effects with reasonable confidence. 


Evaluation objectives 


The primary research question was to identify the impact of the Catch Up Numeracy intervention on 
individual pupils over a 30-week intervention period. Pupils’ numeracy skills were measured by 
performance on the Basic Number Screening Test. 


The process evaluation explored the implementation and scalability of the intervention. 


Project team 


The internal evaluation team was led by Dr Ann Dowker of Oxford University and Dr Graham Sigley 
from Catch Up. These evaluators were responsible for the recruitment of schools and TAs, the training 
of TAs to use the Catch Up intervention, the administration of the numeracy tests and undertaking 
follow-up sessions with TAs at the end of the intervention. 


The external evaluation team at the NFER was led by Simon Rutt, Head of Statistics. While the overall 
project and the impact evaluation were led by Simon Rutt the process evaluation was led by Claire 
Easton, a Senior Research Manager within the Research Department. The NFER team were 
responsible within the impact evaluation for the randomisation of TAs and pupils, the analysis of test 
data and the writing of a final evaluation report. The NFER was responsible for the whole of the 
process evaluation and its contribution to the final report. 


Ethical review 


The NFER has a well-developed Code of Practice that contains detailed ethical protocols. These 
protocols govern all research undertaken by NFER and the trial lies within them. Parents gave active 
written consent for all eligible pupils put forward for the intervention and testing, and the Catch Up 
team confirmed that consent had been received before continuation of the trial. 
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Trial design 


This was a multi-centre, three-arm, parallel-group pupil-randomised trial within schools in England. 
The trial involved 54 schools (two with two test sites within the same school) with two TAs and six 
eligible pupils within each school. Teaching assistants were randomised to one of two intervention 
groups: one delivering a time equivalent intervention and the other delivering the Catch Up 
intervention. The six pupils within each school were randomly allocated to one of three groups: Catch 
Up, time equivalent and control. This design was developed from earlier evaluations that noted a 
Catch Up effect but could not differentiate this effect from the quality of the TA, or the potential bias in 
pupil allocation. The randomisation of TAs and pupils eliminated these potential sources of bias, while 
the time equivalent group of pupils will allow analysis to identify any Catch Up effects over and above 
the effects of one-to-one teaching. 


Eligibility 


Schools were selected and approached by Catch Up prior to the involvement of the external evaluator. 
These were originally planned to be around three main areas: Oxford, Southend and Thurrock. NFER 
therefore had no input into the early phases of the trial set up. Each school paid £275 to take part in 
the trial; this is a little less than the normal cost of Catch Up Numeracy materials and training of £350 
per person. The costs associated with the trial may result in a specific type of school that is 
predisposed to the use of interventions, and one that is prepared to invest in such activities, agreeing 
to take part. However, this recruitment strategy has the advantage of being similar to reality since 
schools were charged a cost. 


The normal intervention process is for eligible pupils to be chosen by the nominated individual after 
they had received training from Catch Up. For the purposes of the trial pupils were selected prior to 
any training. It was agreed that TAs would be able to identify those pupils who were struggling with 
numeracy and would benefit the most from the Catch Up intervention. This occurred before any 
randomisation so is free from bias that could impact on trial findings. 


Intervention 


Catch Up Numeracy is a one-to-one intervention for pupils who are struggling with numeracy. It 
consists of two 15-minute sessions per week which are usually delivered by a trained TA outside of 
the usual teaching class. To prepare them for delivering the intervention, TAs are supplied with 
detailed session plans and receive three half-day training sessions. The intervention breaks numeracy 
down into ten components, assesses children’s ability on each, and targets subsequent instruction so 
that the tutor always addresses the exact area of weakness. 


The intervention is made up of four key stages: assessments for learning, identifying an appropriate 
focus, individual sessions and ongoing monitoring. These are outlined below. 


Stage one: assessments for learning 


e abank of easy-to-administer assessments to determine what the learner can do and where 
their needs lie 


e setting the learner's level and identifying the appropriate starting point for the intervention. 
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Stage two: identifying an appropriate focus 


e using the results of the assessments for learning to complete the Catch Up Numeracy learner 
profile 


e using the learner profile to set the target level and identify an appropriate focus for numeracy 
intervention. 


Stage three: individual sessions 


The trial instructed TAs to deliver two 15 minute sessions each week for a 30 week period. These 15 
minute sessions break down into three sub parts; 


Stage four: ongoing monitoring 
The learner profile is revisited and the Catch Up Numeracy target level is reviewed. 
The review and introduction: 


e reminds the learner of what was achieved in the previous session and outlines the focus of the 
current session 

e confirms to the learner the number range being used 

e introduces and reviews key vocabulary 

e _ links the focus of the current session to the learner’s mainstream class teaching, where 
appropriate. 

The numeracy activity: 

e gives an opportunity for the learner to work on the focus of the session in a range of learning 

styles; e.g. Visual; Aural; Verbal; Physical 


e familiarises the learner with vocabulary appropriate to the focus of the session. 


The link recording: 


e reinforces the skills learned during part two of the individual session, using writing/recording 
as an additional approach to learning 


e provides focused teaching based on observed miscues (i.e. mistakes). 
Below is a description of one of the Catch Up Numeracy activities. 
Cool fans! 


Tell the learner that the number fan is very good at making numbers but needs help to keep the 
numbers in order. 


Use the fan to make a 1, 2, or 3 digit number as appropriate for the number range. Write the 
number on the whiteboard. Repeat for two further numbers 


Ask the learner to point to the numbers in order of size, smallest first. 


Repeat the activity for other sets of 3 numbers 


Repeat the activity, asking the learners to point to the numbers in order of size, starting with the 
largest number. 


Education Endowment Foundation 10 


Methodology 


Pupils were assigned to one of three groups as part of the trial: a pure control group that received 
normal teaching; an ‘equivalent time’ intervention group that received two 15-minute sessions a week 
to replicate the one-to-one nature of the intervention; and a Catch Up intervention group that 
received the intervention as described above. The ‘equivalent time’ group could receive any form of 
numeracy support as long as it was not Catch Up and the TAs were asked to keep a log that recorded 
the actual focus of their sessions. It became evident during the process evaluation interviews and from 
manipulation check questionnaires that the some of the ‘equivalent time’ TAs deviated from the 
original delivery protocol. The interpretation section in the conclusion discusses this issue in more 
detail. 


Outcomes 


The primary outcome measure focused on numeracy ability, as measured by the Basic Number 
Screening Test®. Secondary outcome data was collected by Catch Up and the University of Oxford on 
reading ability, using the Salford Sentence Reading Test® , and on general ability, using the Non- 
Reading Intelligence Tests 12, 


Dr Dowker and the Catch Up office recruited research assistants from among students at the 
University of Oxford to administer the tests. Dr Dowker trained the research assistants in test 
administration; they were also provided with the necessary test manuals. 


The Catch Up office contacted each school and agreed a suitable date and time for a research 
assistant to visit the school and administer the tests. The visits were organised on a geographical 
basis in order to minimise costs and the test schedule took place in as short a time period as possible. 
Pre-tests were administered between September and November 2012 with post-tests administered 
between June and July 2013 The research assistants then visited the schools and administered the 
tests. For the pre-intervention tests, the research assistant did not know which of the three groups the 
pupils had been assigned to and did not know who they were testing until they arrived at the school. 
Where a pupil was absent, arrangements were made for a research assistant to re-visit the school, 
where possible. In a small number of cases, a Catch Up approved trainer who had no connection with 
this trial administered tests to absentees. These trainers also administered the tests with no 
knowledge of group membership. A similar process was followed for the post-intervention re-testing. 


All test papers were sent to Dr Dowker who marked all the papers and generated the raw scores, 
subsequent maths ages and standardised scores. Dr Dowker did not know the membership of each 
group until she had finished marking the post-test data. All data was sent to NFER via a secure data 
transfer portal. 


The process of test administration and test marking was not evaluated by the external EEF evaluator 
but there is no evidence to suggest that this has not been conducted by following appropriate 
protocols. 


? http://www.hoddertests.co.uk/tfsearch/numeracy-maths/bnst.htm 
3 http://www.hoddertests.co.uk/tfsearch/reading/nssrt.htm 


* http://www.hoddertests.co.uk/tfsearch/reasoning/nrit1-3.htm 
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Sample size 


The sample size was determined by the intervention provider prior to the external evaluators (NFER) 
being appointed. 


NFER carried out power calculations on the basis of the planned sample size before the trial 
commenced. In calculating the power of the design, intra-class correlation (rho) was assumed to be 
0.05. As pupils were being randomised within schools this was assumed to be negligible but still 
present. Given the sample sizes and assumptions on rho and looking for a power of >0.80 this design 
should be able to detect an effect of at least 0.25. Evidence from earlier studies suggests that this 
design is well powered. 


100% 


Power 
a 
So 
S 
S 


0.00 0.05 0.10 0.15 0.20 0.25 030 035 O40 0.45 0.50 


Effect size 


—N=100 ——N=100  --=-N=100 


Randomisation 


NFER was responsible for the randomisation of TAs to the Catch Up delivery and equivalent time 
groups as well as the randomisation of pupils to one of the three trial groups. Schools were sent an 
Excel spreadsheet to populate details of TAs and the six eligible pupils. Schools first sent the list of 
two TAs and these were randomly allocated to either deliver the time equivalent sessions or the Catch 
Up Numeracy sessions. These randomisations were returned to the schools and the selected TA 
received the required training. 


Schools selected six eligible pupils and this list was sent to NFER for randomisation. Two pupils were 
randomly allocated to each of the three groups: control, time equivalent and Catch Up. The 
randomisation process was run for each school separately. The schools Unique Reference Number 
(URN) was used as the seed for the random number generator used to randomly allocate pupils and 
TAs. Randomised lists were returned to schools 
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Analysis 


Analysis was undertaken on an intention to treat (ITT) basis. This was a pupil-randomised trial but the 
randomisation was stratified by school. Since school has therefore to be taken into account, analysis 
was undertaken using a multi-level model. Data was prepared in SPSS, using each school’s URN as 
the cluster identifier; analysis was undertaken in the R statistical package using a two-level model 
(pupil and school). Pre-intervention test scores were used as a covariate to the post-intervention test 
score. Two dummy variables were included in the analysis to identify pupil membership of either the 
control group or Catch Up intervention group. This allowed analysis to identify differences between the 
equivalent time group and the control group and between the Catch Up intervention group and the 
equivalent time group. 


Sub-group analysis introduced variables for eligibility for free school meals (FSM) and gender. This 
helped determine if effects were the same for all pupil characteristics or if there was a differential 
effect. Further terms were introduced looking at the interaction between FSM and group membership 
and with gender and group membership. Interaction terms that incorporated pre-intervention test 
scores were additionally introduced into the models. This sub-group analysis was pre-specified in the 
trial protocol. 


The analysis described above was replicated on the secondary outcomes using pre-intervention and 
post-intervention test score data obtained from the Salford Sentence Reading Test and the Non- 
Reading Intelligence Tests 1-3. 


An analysis specification was developed by the lead external evaluator and passed to a statistician 
within NFER who undertook analysis without knowing the group membership conferred by the dummy 
variables. 


Process evaluation methodology 


The process evaluation comprised three phases. In addition to these three phases manipulation check 
questionnaires were also sent to TAs. While these are primarily part of the impact evaluation they will 
be reported in the process valuation section as the outcomes are similar. 


Firstly the team attended and observed the ‘Delivering Catch Up Numeracy’ and ‘Managing Catch Up 
Numeracy’ training sessions prior to the intervention commencing. The team attended training 
sessions in four locations during September and October 2012. Phase two involved observing the 
follow-up (post-intervention) ‘Review and Next steps’ sessions in the same locations during June and 
July 2013. The team was keen to observe training delivered across locations and carried out by 
different trainers to assess the extent to which the training followed the same pattern. 


For the third and final phase, the team undertook telephone interviews with 25 TAs and 24 Catch Up 
Numeracy coordinators across 27 participating schools (N=49 participants). Each school has a Catch 
Up Numeracy coordinator who oversees the intervention within the school and offers support to the 
TAs. Researchers chose this data collection method in order to gain in-depth qualitative data from a 
wide range of interviewees during a short timescale while ensuring disruption to the participating 
schools, TAs and coordinators was kept to a minimum. This was particularly important given the timing 
of the interviews: the end of the academic year. 


Originally the team randomly sampled one TA or one coordinator from each participating school and 
invited them to participate in the research. However, due to a low response rate to the original 
invitation, the team revised the strategy and invited all TAs and coordinators to participate in an 
interview with the aim to achieve up to 50 participants in total. The timing of the interviews meant that 
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recruiting TAs and coordinators was challenging: furthermore their availability was limited due to end 
of school- year activities. 


Two experienced NFER researchers carried out all interviews via telephone. Interviews took between 
15 to 40 minutes, with most taking around 25 minutes. As per NFER’s Code of Practice, at the start of 
every interview the researchers explained the purpose of the interview, and the data protection and 
confidentiality protocols. Interviews were recorded only where interviewee permission was granted. 
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Impact evaluation 


Timeline 


The intervention commenced at slightly different times in each school due to issues of school 
recruitment and the training of TAs. The first pre-test was carried out on 26 September 2012 with the 
last on 23 November 2012. Pupils were tested before the commencement of intervention delivery and 
at the end of the 30 week delivery period. Schools were contacted by the Catch Up office to arrange 
convenient times for research assistants to visit and conduct the tests. For post-testing this occurred 
as close as possible to when the intervention was completed within the schools. 


Participants 


Schools were recruited to the trial in one of three ways. 


e Any school that contacted Catch Up between April and September 2012 enquiring about 
Catch Up Numeracy was offered the opportunity to be part of the project if they could get other 
geographically neighbouring schools involved (in order to create a viable training group). 


e Catch Up local authority contacts were asked if they had schools that were in challenging 
contexts and that could benefit from intervention support. 


e Catch Up directly approached some schools. 


Approximately 130 schools were approached by Catch Up with 54° agreeing to take part in the trial. 


All interested, and potentially interested, schools were invited to briefings® and provided with a project 
briefing paper. Schools that could not get to a briefing were followed up by email and telephone. 
Participating schools were asked to complete an evaluation agreement and schools were recruited up 
to October 2012 half term. 


Eligible pupils were identified by the schools prior to the training using the school’s own assessment 
of need. This approach sought to avoid any contamination which could arise from the trained TAs 
working with all six pupils in each school. This process is a variation from the normal selection of 
eligible pupils for the Catch Up intervention. 


The tables below identify the characteristics of the schools taking part in the trial. Table 1 identifies 
that schools were predominantly drawn from Wales and the southern half of England. Table 2 
highlights the quintiles of pupils with eligibility for free school meals within these schools. It shows that 
approximately 44% of schools had higher levels of free school meal eligibility while only 26% of 
schools came from the lower levels of eligibility. Table 3 identifies the extent of special educational 
needs (SEN) within trial schools. Approximately 76% of trial schools had more than 16% of their 


° Two schools had two sites and were treated separately. They provided six pupils and two TAs from 
each school/site thus resulting in the total sample of 336 pupils. 


® These were not observed by NFER as they occurred before the external evaluator was appointed. 
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school cohort assessed as having SEN, which compares to approximately 65% in the overall 
population of primary schools. 


Table 1. Geographical location 


Midlands 25 46.3 46.3 
South 18 33.3 79.6 
Wales 11 20.4 100.0 
Total 54 100.0 


Table 2. Eligibility for free school meals (quintiles) 


Lowest 20% 3 5.6 5.8 5.8 
2nd lowest 20% 11 20.4 21.2 26.9 
Middle 20% 14 25.9 26.9 53.8 
2nd highest 20% 12 22.2 23.1 76.9 
Highest 20% 12 22.2 23.1 100.0 
Total 52 96.3 100.0 

N/A 2 3.7 

Total 54 100.0 


Table 3. Percentage of pupils with special educational needs (banded) 


0-5% 0 0 0 0 
6 - 10% 2 3.7 3.8 3.8 
11-15% 10 18.5 19.2 23.1 
16 - 24% 21 38.9 40.4 63.5 
25% + 19 35.2 36.5 100.0 
Total 52 96.3 100.0 
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N/A 2 3.7 


Total 54 100.0 


At the time of analysis, the Key Stage 2 results of 2012 were only available for 39 schools (75%) 
schools within the trial. 27% of these schools were in the top two quintiles of Key Stage performance 
and 41% were in the bottom two quintiles. 


Forty-three of the trial schools had recent OFSTED judgements. 63% (27) of these were considered 


‘Good’, 21% (9) were considered ‘Satisfactory’, 12% (5) were considered ‘Outstanding’ and 5% (2) 
were considered ‘Unsatisfactory’. 
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Assessed for eligibility (n=) Excluded (n=) 


Not meeting inclusion 
criteria (n=) 


Declined to 
Randomised (n=336 pupils, n=112 participate (n=) 
TAs) 


Other reasons (n=) 


Control Group Time Equivalent Group Catch Up Numeracy Intervention 


Allocated to control group (n=112 pupils) lceaon Allocated to intervention group (n=112) Allocated to intervention group (n=112) 
Allocation 
Received allocated intervention (n=112) 


Received allocated intervention (n=112) Received allocated intervention (n=112) 


Did not receive allocated intervention Did not receive allocated intervention (n=0) Did not receive allocated intervention (n=0) 


(n=0) 


Lost to follow-up (n=1) 


Lost to follow-up (n=0) Lost to follow-up (n=0) 


Discontinued intervention (n=0) Follow-Up 


Analysed (n=108) 


Follow-Up 


Discontinued intervention (n=5) Discontinued intervention (n=0) 


Analysed (n=102) Analysed (n=108) 
Analysis Aue 
Excluded from analysis (n=3) 


Excluded from analysis (n=4) Excluded from analysis (n=5) 
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The sample information reported on the previous page would normally include details of the number of 
pupils assessed for eligibility. The external evaluator was appointed to this trial after this event had 
occurred and so had no control over how it was undertaken, although there is no reason to doubt that 
it was undertaken in an appropriate manner. Unfortunately details of the number of pupils approached 
have not been maintained. The table identifies how many pupils were excluded from the ITT analysis. 
In the control group four pupils were excluded, one due to absence from the pre-test and three due to 
absence from the post-test. In the time equivalent group five pupils left the school and were not 
contactable for a post-test, three pupils were absent for the pre-test and two were absent from the 
post-test. In the Catch Up intervention group four pupils were absent for the post-test. The overall 
attrition was 18 pupils, 5.4%’. 


Pupil characteristics 


Tables 4 and 5 below highlight pupil characteristics by each of the three groups within the trial. 
Although outcome test scores were age standardised the age profile can also be identified. The 
overall age mean was 97.7 months, or just above 8 years old. 


Table 4 Gender by randomised group 


Control 54.5% 45.5% 112 
Time equivalent 39.1% 60.9% 110 
Catch Up 50.0% 50.0% 112 
All pupils 47.9% 52.1% 334 


Table 5 Eligibility for free school meals by randomised group 


Control 67.0% 33.0% 112 
Time equivalent 68.8% 31.3% 112 
Catch Up 64.0% 36.0% 111 
Total 66.6% 33.4% 335 


’ This attrition rate varied between the three groups. For the control group and the Catch Up® 


intervention group it was 3.2% and for the time equivalent intervention group it was 8.9%. 
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Outcomes and analysis 


The main analysis is an ITT analysis and the results of that analysis are presented in Table 6 below. 
See the Technical Appendix for a description of how the analysis was undertaken 


Table 6 — Intention to Treat Analysis Outcomes 


Intercept 39.29 4.76 29.96 — 48.62 0.00 
Maths Pre-test score 0.63 0.06 0.51 —0.75 0.00 
Control group -3.54 1.40 -6.28 — -0.80 0.01 
Catch Up group -0.79 1.40 -3.54 — 1.96 0.57 


The main analysis was to determine if there was a significant difference in amount of progress made 
by pupils in the Catch Up intervention group when compared to those pupils in the control group. To 
account for the potential effects of one-to-one teaching a time equivalent group was introduced to the 
design. This group of pupils acted as the default group of pupils and analysis identifies any differences 
between this group and the control group, and this group and the Catch Up intervention group. Pupils 
in the control group, on average, had a post-test score that was 3.54 points below that of the average 
pupil in the time equivalent group. Pupils in the Catch Up intervention group, on average, had a post- 
test score that was 0.79 points below that of the average pupil in the time equivalent group. The model 
controls for the pre-intervention test scores collected before the start of the trial 


Table 6 shows that there is a significant difference in the post-test maths standardised score, 
between the pupils in the control group and those in the time equivalent group, but there is no 
significant difference between the time equivalent group and the Catch Up group. 


The effect size for the significant effect identified above is 0.27 (CI 0.49 — 0.06), based on Cohen's d 
adjusted for small sample sizes using Hedges’ g. While there is no significant difference between the 
two treatment groups a post-hoc test was carried out to determine any significant difference between 
the Catch Up group and the control group. This was found to be significant, p=0.047°. Its effect size is 
0.21 (CI 0.42 — 0.01). 


Sub-group analysis was carried out on the ITT dataset introducing gender, eligibility for free school 
meals and a number of interaction terms. Interaction terms were created for pre-intervention test 
scores and gender, pre-intervention scores and fsm, pre-intervention scores and control group 
membership and pre-intervention scores and Catch Up membership. Table 7 reports this analysis. 


8 Bonferroni adjustment has not been applied 
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Table 7 Sub-group analysis outcomes 


Intercept 37.69 9.18 19.70 - 55.68 0.00 
Maths pre-test score .64 .10 0.44 - 0.84 0.00 
Control group -3.30 2.37 -7.95 — 1.35 0.17 
Catch Up group 1.42 2.36 -3.21 - 6.05 0.58 
Female 1.98 2.17 -2.27 — 6.23 0.36 
FSM - Yes -2.72 2.37 -7.37 — 1.93 0.25 
Female in control group -2.29 2.97 -8.11 — 3.53 0.44 
Female in Catch Up group -5.03 3.04 -10.99-0.93 0.10 
FSM in Control group 2.74 3.19 -3.51 — 8.99 0.39 
FSM in Catch Up group 1.47 3.28 -4.96 — 7.90 0.66 
Pre-test Score* control group 04 13 -0.21 — 0.29 0.73 
Pre-test Score*Catch Up group AM. H3 -0.08 -0.42 0.18 
Pre-test Score* female -.17 11 -0.39 — 0.05 0.12 
Pre-test Score* FSM -.09 13 -0.34 — 0.16 0.49 


Table 7 does not alter the initial ITT analysis that shows the intervention groups have significantly 
higher post-test scores than the control group. The analysis here does not identify any significant 
interactions. 


Further analyses were run to omit pupils where TAs had altered the delivery of the intervention. These 
were primarily pupils who had received their intervention from a non-randomised TA or where the 
school dropped out of the intervention. This occurred for pupils in both intervention groups and 
resulted in seven exclusions due to the school withdrawing from the trial in February 2013, seven 
exclusions due to a non-randomised TA replacing a randomised TA and four exclusions due to the 
incorrect TA attending the Catch Up training. 


For schools that dropped out of the trial, the control group pupils remained within the analysis as they 
continued to receive the same treatment. The above exclusions resulted in analysis being carried out 
on 108 pupils within the control group, 91 from the time equivalent group and 101 from the Catch Up 
group. The main output from this analysis is in Table 8. 
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Table 8 — Model outcomes for ITT dataset with omissions 


Intercept 40.34 4.98 30.58 - 50.10 0.00 
Maths Pre-test Score 0.62 0.06 0.50 — 0.74 0.00 
Control Group -3.62 1.48 -6.52 - -0.72 0.02 
Catch Up Group -0.90 1.51 -3.86 — 2.06 0.55 


Sub-group analysis on this dataset did not significantly alter the original ITT interpretations. 


Secondary outcomes analysed included pre and post-test data from the Salford Sentence Reading 
Test and the Non-Reading Intelligence Tests 1-3. Data was analysed in the same way as that used for 
the main ITT analysis and while significant increases were found in the post-test scores there was no 
significant difference between pupils in the control group and either of the two intervention groups. 


Cost 


The external evaluators did not explicitly ask about or collect data on the costs associated with the 
intervention. The Catch Up Numeracy intervention costs schools £350 per person trained. As with any 
intervention, additional resources are required to ensure its successful delivery. For Catch Up 
Numeracy, these include: 


e arranging cover for each staff member to attend the different training sessions (a minimum of 
two days in total) 


e time to plan or prepare for the session with pupils. Some TAs noted that once the resources 
and materials were set up and TAs became familiar with the intervention, less time would be 
needed but that some planning time is still required 


e time to access the website or download resources. 


A very small number of TAs noted that photocopying is required to deliver the intervention, for 
example the progress booklets and individual session sheets. Making these available on A5 paper or 
adding more than one session to each sheet was suggested as a way to reduce the amount of paper 
needed. The costs associated with these different activities have not and cannot be quantified as part 
of this project but it is worth noting that additional resource input is required. 


Independent research reported by New Philanthropy Capital (Paterson ef a/, 2010) estimated that the 


cost of Catch Up Numeracy is £120 per pupil (increased to £130 on the Catch Up website) which 
accounted for the required resources and TA time. 
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Implementation 


Despite Catch Up making the request, five coordinators involved in the interviews did not attend the 
three half-day N2 training sessions; however all TAs interviewed had attended. In contrast only 11 of 
the coordinators interviewed attended the N4 training session at the end of the intervention. Most 
found the training incredibly useful, detailed and delivered to an appropriate pace. A small number of 
interviewees (four coordinators) commented that it can be expensive for a school to have a senior 
leader and a TA out of school for three half days, but they recognised the value of the model whereby 
the theory is given and can be applied in school during the afternoons. It became apparent during the 
observations that not all TAs returned to school during the afternoon to practise Catch Up Numeracy; 
however, most TAs had practised with a family member or friend instead. 


When asked if they had to spend any additional time familiarising themselves with the Catch Up 
Numeracy materials after the training and before working with the learners, 18 TAs said yes. This 
seemed to vary from a quick ‘ten-minute’ read through the materials to a more thorough revision 
taking ‘several hours’. Seven coordinators said they spent some time doing this but most had not. 


The additional support Catch Up provided to the project schools was highly valued by TAs and 
coordinators alike. They felt that having a trainer visit and support them helped clarify any issues, 
ensured practice remained focussed, and provided affirmation. For a small number of TAs (three), the 
Catch Up trainer observations also improved practice (mostly around completing the record sheets). 


TAs and Catch Up coordinators identified a number of barriers to delivering Catch Up Numeracy, 
relating to various issues. 


e Alack of time, specifically around TAs having sufficient time to plan and prepare for the 
sessions and also finding time to fit in two 15-minute sessions each week within the existing 
school timetable. This was mentioned by ten TAs and 12 coordinators. 


e Regular pupil absence made it difficult to do all 60 sessions. This was mentioned by seven 
TAs and three coordinators. 


e The negative attitude of some pupils towards the intervention, mentioned by five TAs and two 
coordinators. For example, pupils resented being singled out as ‘not very good at maths’ or 
being taken out of lessons they enjoy, such as PE, in order to do Catch Up. 


e The difficulty in finding suitable resources to run the intervention. This was mentioned by six 
TAs and five coordinators. 


e = The difficulty in finding a suitable location within school to run the sessions. This was 
mentioned by two TAs and three coordinators. 


e Alack of senior management support and commitment to the intervention. This was 
mentioned by four TAs and five coordinators. 


Fidelity 


No clear patterns emerged when schools ran the Catch Up Numeracy sessions. Sessions were held 
before school where learners were offered breakfast (in one case); during registration or assembly 
times (four cases); morning lessons, including during the last 15 minutes before lunchtime (four 
cases); at lunch time (in one case); during afternoon lessons (because core lessons were held during 
the morning - this was mentioned seven times) or during maths lessons (in one case). For others, it 
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varied each week (seven cases). Some schools had ring-fenced time for Catch Up Numeracy to be 
held but this was not commonplace. Around half of schools had a policy about when interventions 
should be run - for example, not during mornings when core subjects were taught. It was during this 
time, that Catch Up sessions were held. 


Very few interviewees reported that learners had received two 15-minutes sessions each week. TAs 
tried to adhere to the intervention where possible, but pupils or staff absences prevented this from 
happening. Furthermore they reported scheduling and school-year activities (such as trips and 
assessments) resulting in Catch Up sessions not happening. 


When asked if the sessions tended to last for 15 minutes each, most TAs (19) and coordinators (17) 
said they did. However, several noted that, particularly during the early stages, the sessions tended to 
exceed 15 minutes; for some, this happened throughout the intervention but was usually only 
extended by a few minutes. TAs noted that learners asking questions and writing their thoughts in the 
individual record sheets meant that time overran a little. Three TAs noted that they spent 15 minutes 
with the learners but that Catch Up Numeracy required additional TA time for preparation and 
planning. 


There were several elements of the intervention which TAs and coordinators considered to be 
adaptable. The most common one was the length of time the intervention was run for. Eight TAs and 
six coordinators suggested that this could be flexible to the needs of the pupil rather than having to be 
30 weeks’ (the length of time for this project). 


Four TAs and coordinators commented they thought that Catch Up Numeracy did not have to be 
delivered on a one-to-one basis but could be delivered to small groups instead. Three coordinators 
commented that the amount of paperwork that was required to be completed, particularly around 
recording and monitoring pupil progress could be adapted. The other element of the intervention that 
was perceived as being adaptable was the number of sessions run per week, with two coordinators 
commenting that this did not necessarily need to be two per week. 


Outcomes 


Pupil outcomes 


Almost all TAs delivering the Catch Up intervention (20) and the school coordinators (23) identified an 
improvement in pupil confidence towards numeracy. This was evidenced by pupils participating more 
fully in numeracy lessons and being more willing to put their hand up and answer questions in class. In 
a number of cases TAs and coordinators commented that these outcomes were not just restricted to 
numeracy but had been transferred more widely to other lessons also. 


Other outcomes that were mentioned by both TAs and coordinators alike included the following. 
e Pupils becoming more engaged in numeracy. This was mentioned by three TAs and five 
coordinators. 


e Improvements to pupil attitude and self-esteem and a greater enjoyment and enthusiasm 
towards numeracy. This was mentioned by eight TAs and seven coordinators. 


e Pupils being better behaved, more focussed and less disruptive in class as a result of 
Catch Up. This was mentioned by three TAs and three coordinators. 


: Usually schools would carry out Catch Up with pupils for as long as they felt it was needed. However, 
for the purposes of this project each pupil received the intervention for 30 weeks. 
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e Pupils becoming more reflective and more aware of their strengths and weaknesses in 
numeracy. Mentioned by two TAs and two coordinators. 


e Insome instances there were changes to how pupil’s approached their learning, for example 
pupils were more willing to take ownership of their work. This was mentioned by nine TAs and 
seven coordinators. 


In terms of the perceptions on the impact of the intervention on pupil attainment in numeracy there 
was a range of views. Coordinators were more inclined to comment on the attainment outcomes of 
Catch Up in terms of national curriculum levels and test scores than TAs, with 11 coordinators 
mentioning this in contrast to only six TAs. Some coordinators and TAs perceived there to be big 
increases in pupils’ understanding in maths and in their attainment. In other cases schools felt that 
pupils were making the expected two sub-levels of progress while in a small number of cases TAs and 
coordinators were disappointed with the progress that some pupils were making in relation to 
attainment. 


Ten TAs and nine coordinators identified the one-to-one nature of Catch Up Numeracy and its 
personalised approach to the individual pupil as the reasons behind any changes to attainment 


Impacts on TAs and coordinators 


TAs expressed satisfaction in delivering Catch Up Numeracy and many (ten) mentioned the 
confidence they had gained from being given the responsibility of running the intervention in school. 
Six TAs also commented that they had gained a greater insight and awareness of pupils’ strengths 
and weaknesses in maths as a result of delivering Catch Up. 


Twelve coordinators also commented on the professional development benefits for TAs. Coordinators 
reported fewer outcomes for themselves; these included increased awareness of how to support 
pupils with numeracy, fresh ideas about numeracy activities that could be used in school and a greater 
understanding of gaps in pupil learning (particularly facilitated by the Catch Up Numeracy assessment 
activities). In four cases coordinators commented that the intervention had very little impact upon 
them. 


A number of other consequences were reported by TAs and coordinators, some positive and some 
negative. The two most common positive consequences related to improved relationships between 
staff and project pupils, mentioned by five TAs and one coordinator, and closer working and better 
communication between different members of school staff (class teachers, TAs, Catch Up 
coordinators and SENCOs), this was mentioned by three TAs and two coordinators. 


TAs and coordinators reported negative effects of Catch Up relating to increased workload, and 
pressure and stress associated with trying to plan, prepare and deliver Catch Up Numeracy in an 
already busy timetable. In some instances these negative effects were exacerbated as a result of 
participation in this project due to its requirement to complete 60 sessions with pupils over the course 
of 30 weeks. 


TAs and coordinators said the best aspects of Catch Up Numeracy were: 


e the confidence it gives pupils, in maths but also often across the curriculum 


e the assessment, which is diagnostic and means the intervention is tailored to learners’ 
individual needs 


e its structure, including the resources such as the progress sheets 


e the one-to-one support learners are given. 
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Formative findings 


TAs and coordinators suggested ways in which the intervention could be improved, several of which 
related to the Catch Up website and included the following. 


e There needs to be more resources and linked recording activities available on the website 
which could be used by TAs. This was mentioned by six TAs and five coordinators. 


e The resources on the website need to be updated so that they are more modern. This was 
mentioned by two TAs and two coordinators. 


e The navigation through the website could be made easier. This was mentioned by three TAs 
and two coordinators. 


A number of TAs commented that they felt that the initial assessments could be improved in the 
following ways. 


e By making them more flexible so that the TA does not have to immediately stop the 
assessment if the pupil makes a mistake. Mentioned by three TAs. 


e By simplifying the language used in the assessments so that it is simpler and more child 
friendly. Mentioned by three TAs. 


A number of coordinators felt that Catch Up could be improved by reducing the amount of paperwork 
involved in delivering and administering the intervention. Furthermore several coordinators suggested 
that Catch Up could be improved by running it in small groups rather than one-to-one, or by running it 
for a shorter period of time than 30 weeks. 


While most coordinators recognised that Catch Up Numeracy should not necessarily link with normal 
numeracy teaching, TAs reported that they linked the Catch Up sessions where they could with class 
teaching. This was not possible in some instances, for example where the number range in class was 
higher than the focus of Catch Up. Dialogue between the TAs and the class teacher seemed to vary 
between schools. Some TAs felt dialogue was important, whereas for others it was more difficult as 
they worked with different year groups for Catch Up than they usually would. A small number of 
interviewees said that in future they would like the class TA to deliver Catch Up so where linkages 
can be made this would be possible. 


Teaching assistant questionnaires 


To help in the understanding of how TAs delivered the two interventions, two questionnaire surveys 
were carried out on a sample of TAs at the mid-point of the 30 week trial period and at the end. 
Questions were asked about how the training matched up to reality and how the TAs within each 
school managed their respective parts of the trial and whether there was any potential for cross 
contamination in the support provided to the pupils. Catch Up TAs were also asked about how they 
delivered the Catch Up intervention. Many of the results from the survey follow what was discovered 
during the process evaluation. 


The results of the two questionnaires are discussed below but, to summarise, there would seem to be 
some discrepancies between how the trial protocol instructed the TAs to deliver their part of the 
intervention and what happened in practice. In asking teachers to deliver a number of sessions per 
week, for a set number of weeks, was potentially always going to result in a variation in delivery, 
primarily due to timetabling issues and TA and pupil absences. 
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In looking at the number of sessions undertaken there is no significant difference in the reported 
number of sessions the TAs delivered. Teaching assistants delivering the Catch Up intervention also 
broadly delivered it as expected and as the training had instructed. To maintain fidelity to the trial the 
delivery of the time equivalent sessions required TAs not to deliver Catch Up but to keep a log of the 
activities they did deliver. Questionnaire responses identified below would seem to indicate that a 
proportion of TAs delivering the time equivalent sessions had more knowledge of Catch Up Numeracy 
than could be thought of as ideal. Additionally, some TAs adjusted the delivery of their sessions based 
on this knowledge. 


Time equivalent TAs returned 32 questionnaires at the mid-point of the trial. Twenty-five (78%) of 
these TAs reported that they held two sessions per week, while four TAs (13%) reported that they held 
four sessions per week. In the second survey, of the 33 respondents, 28 (85%) said they had followed 
the protocol of two sessions per week. As for the length of these sessions, 26 TAs reported that 
sessions lasted between 15 and 20 minutes. In the second questionnaire this had remained consistent 
for the 27 respondents (84%). Although a desired result would be for all TAs to follow the agreed 
protocols exactly, the workings of a busy school probably make this difficult (as the process evaluation 
team found out). Given the responses above it would appear that a high proportion of time equivalent 
TAs delivered their sessions as planned. A potential cause for concern is that, even though they did 
not receive the Catch Up training, 15 time equivalent TAs (47%) reported at the first survey that they 
did know a little about the Catch Up approach to teaching numeracy. Thirteen TAs (41%) at the 
second survey reported similar knowledge. If this knowledge led to teaching methods being more 
similar to those adopted by the Catch Up TAs then this makes it harder to maintain a separation of 
treatments. To further complicate this issue, 12 time equivalent TAs (38%) at the first survey and nine 
TAs (31%) at the second survey reported that their approach to supporting pupils had changed based 
on what they knew about Catch Up Numeracy. 


For the Catch Up TAs there were 40 respondents to the mid-point survey and 35 to the second survey 
at the end of the trial. A high proportion again reported that they had held two sessions with their 
pupils, 35 (86%) at the mid-point survey and 27 (77%) at the second survey. They also reported that 
their sessions were between 15 and 20 minutes in length, 35 (86%) at the mid-point survey and 33 
(97%) at the second time point. 


Teaching assistants for the time equivalent and Catch Up groups were required to complete logs 
recording the sessions they had held with their pupils and an investigation of these shows that for 74 
pupils in the time equivalent group, the mean number of sessions received was 35.5. For 76 pupils in 
the Catch Up group, the mean number of sessions was 34.9. This difference is not statistically 
significant. While we are unable to determine the length of these sessions it would seem to be a long 
way off the 60 sessions in the original protocol, although both treatment groups appeared to have 
received a similar number of sessions. 


At the mid-point survey, when asked how well equipped they felt to deliver Catch Up, only 21 (53%) of 
respondents said to a great extent or entirely. This rose to 27 (77%) respondents at the second 
survey. This may identify some variable delivery of the Catch Up intervention. Other possible 
identifiers of variable delivery are from the question that asked if the TAs had adapted the Catch Up 
Numeracy intervention. At the mid-point survey six (15%) of respondents said they had to ‘a great 
extent’ or ‘entirely’, while a further 15 (38%) said they had adapted the intervention ‘to some extent’. In 
the second survey six respondents said they had adapted the intervention ‘to a great extent’ and a 
further 18 (51%) said they had adapted it ‘to some extent’. These responses, from 66% of all Catch Up 
TAs, would suggest that the Catch Up intervention may not have been delivered as intended. 


Given the results of the trial described above it is interesting to also note that when asked if pupil 
enjoyment and confidence in maths had increased, 25 (70%) of TAs said that it had ‘to a great extent’ 
or ‘entirely’. When asked if it had improved attainment in maths only 14 (41%) of the TAs reported that 
they felt it had to ‘a great extent’ or ‘entirely’. 
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Limitations 


The main limitation of the study is with the potential cross-contamination between TAs delivering the 
time equivalent sessions and those delivering the Catch Up sessions, and the degree to which this 
has impacted on the results. Other more minor threats to the internal validity of the trial come from the 
level of attrition with a number of pupils dropping out of the analysis, although the overall percentage 
of attrition is small when compared to other trials. Attrition is also at the same rate for the control group 
and the Catch Up group. Testing was organised and managed by the delivery organisation with 
research assistants being trained to administer the tests, and although there was no independent 
monitoring of this process there is no evidence to suggest that the administration of the tests, or their 
marking, were not undertaken following accepted protocols. Test administration and marking for both 
the pre and post-tests were, so the external evaluator was informed, carried out blind to group 
membership. There is no evidence to suggest that this was not the case. 


Interpretation 


The main hypothesis that this trial was designed to test was whether the gains in numeracy 
attributable to those pupils who had been randomly assigned to the Catch Up group were significantly 
greater than the gains made by pupils who had been assigned to the group that had received sessions 
that were time equivalent. Both of these groups would have been expected to make gains significantly 
greater than the control group as the effectiveness of one-to-one tuition is well documented (see 
Sutton Trust-EEF Teaching and Learning Toolkit). 


We conclude that this trial has seen no evidence that Catch Up has produced significant gains in 
numeracy over and above those that were seen in the time equivalent group. The Catch Up group had 
a gain that was significantly greater than the control group. But, the conclusion from this trial is that 
while there is evidence that one-to-one sessions, over a sustained period, do produce a significant 
improvement in numeracy skills, there is no evidence that Catch Up adds anything over and above 
this one to one teaching effect. 


As described in the process evaluation, and from what can be determined from the questionnaire 
responses, there would appear to be some variance in how TAs delivered the sessions to their relative 
groups. It is not clear whether these differences in delivery have actually impacted on how pupils have 
developed their skills in numeracy. We do know that the training of Catch Up TAs was kept separate 
and that TAs assigned to the time equivalent group would not have had access to this training. We 
also know that trainers make it clear that the contents of the numeracy intervention should not be 
cascaded to colleagues and we have to assume schools adhered to this requirement. While it should 
be fully accepted that teachers and TAs within the same school will communicate with each other, and 
there is evidence from the process evaluation and the questionnaires that this did indeed occur, we 
are unclear as to whether this actually had an impact on the results of this trial. 


It is evident from the process evaluation, and the questionnaires, that TAs and Catch Up coordinators 
valued the intervention and believed it had a positive impact on some pupils. These impacts tended to 
be around softer outcomes such as confidence, engagement with learning and willingness to attempt 
numeracy problems. It was also clear from the process evaluation that delivering a structured 
intervention within a school can be problematic due to pupil absences, TAs’ timetables and conflicting 
demands on their time. An intervention such as Catch Up needs to be planned into the timetable from 
the outset, and not part way through the year, as was the case with this trial. This may help schools 
overcome some of these challenges. This would also assist scalability of the intervention and its roll 
out to more schools. 
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Future research and publications 


There is not enough evidence to be certain that the Catch Up Numeracy does or does not improve 
numeracy attainment over and above pupils receiving one-to-one support. This would suggest the 
need for a further trial to ensure there are better controls of how the TAs deliver the intervention with a 
further trial assisting in any future meta-analysis. Further analysis could be undertaken to look at the 
session logs provided by the TAs to determine any similarities in session delivery 
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Technical Appendix 


Analysis was carried using SPSS and R analytical software. Data was prepared in SPSS by one 
statistician who was aware of the data structure and group membership of the three randomised 
groups. Data files created included identifiers for school, a pupil identifier, a standardised score for the 
pre-test maths score, a standardised score for the post-test maths score, a dummy variable to indicate 
membership of the control group and a dummy variable to indicate membership of the Catch Up 
intervention group. The time equivalent group was set as the default group so that differences could 
be identified between membership of the other groups and this group. This was the primary dataset for 
the intention to treat analysis. Hierarchical models were analysed using R software with the schools’ 
URN number identifying the higher level and the unique pupil identifier the lower level. 


Model coefficients and their standard errors are reported in the main body of the report. Effect sizes 
were calculated using Cohen’s d as identified in EEF protocols. The standard deviations used for 
these calculations were 11.579 for the control group, 14.281 for the time equivalent intervention group 
and 14.268 for the Catch Up intervention group. 


Additional sub-group analyses were undertaken following the same methodology and include the 
creation of dummy variables to identify gender, eligibility for free school meals and interactions 
between these variables and pre-test scores and group membership. Model coefficients for this 
additional analysis are reported in the main body of the report. 
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